Recently, the self-supervised pre-training paradigm has shown great potential in leveraging large-scale unlabeled data to improve downstream task performance. However, increasing the scale of unlabeled pre-training data in real-world scenarios requires prohibitive computational costs and faces the challenge of uncurated samples. To address these issues, we build a task-specific self-supervised pre-training framework from a data selection perspective based on a simple hypothesis that pre-training on the unlabeled samples with similar distribution to the target task can bring substantial performance gains. Buttressed by the hypothesis, we propose the first yet novel framework for Scalable and Efficient visual Pre-Training (SEPT) by introducing a retrieval pipeline for data selection. SEPT first leverage a self-supervised pre-trained model to extract the features of the entire unlabeled dataset for retrieval pipeline initialization. Then, for a specific target task, SEPT retrievals the most similar samples from the unlabeled dataset based on feature similarity for each target instance for pre-training. Finally, SEPT pre-trains the target model with the selected unlabeled samples in a self-supervised manner for target data finetuning. By decoupling the scale of pre-training and available upstream data for a target task, SEPT achieves high scalability of the upstream dataset and high efficiency of pre-training, resulting in high model architecture flexibility. Results on various downstream tasks demonstrate that SEPT can achieve competitive or even better performance compared with ImageNet pre-training while reducing the size of training samples by one magnitude without resorting to any extra annotations.

本文介绍了Omnicity，这是一种从多层次和多视图图像中了解无所不能的城市理解的新数据集。更确切地说，Omnicity包含多视图的卫星图像以及街道级全景图和单视图图像，构成了超过100k像素的注释图像，这些图像是从纽约市的25k Geo-Locations中良好的一致性和收集的。为了减轻大量像素的注释努力，我们提出了一个有效的街景图像注释管道，该管道利用了卫星视图的现有标签地图以及不同观点之间的转换关系（卫星，Panorama和Mono-View）。有了新的Omnicity数据集，我们为各种任务提供基准，包括构建足迹提取，高度估计以及构建平面/实例/细粒细分。我们还分析了视图对每个任务的影响，不同模型的性能，现有方法的局限性等。与现有的多层次和多视图基准相比，我们的Omnicity包含更多具有更丰富注释类型和更丰富的图像更多的视图，提供了从最先进的模型获得的更多基线结果，并为街道级全景图像中的细粒度建筑实例细分介绍了一项新颖的任务。此外，Omnicity为现有任务提供了新的问题设置，例如跨视图匹配，合成，分割，检测等，并促进开发新方法，以了解大规模的城市理解，重建和仿真。 Omnicity数据集以及基准将在https://city-super.github.io/omnicity上找到。

语义表示对于视频文本跟踪（VTT）任务具有很大的益处，该任务需要同时对视频中的视频进行分类，检测和跟踪文本。大多数现有方法通过在连续帧中的外观相似性来解决此任务，同时忽略丰富的语义功能。在本文中，我们探讨了具有对语义和视觉表示的对比学习的强大追踪视频文本。相应地，我们介绍了一个具有语义和视觉表示（SVREP）的端到端视频文本跟踪器，它通过在视频序列中利用不同文本之间的视觉和语义关系来检测和跟踪文本。此外，通过轻量级架构，SVREP在保持竞争推断速度的同时实现最先进的性能。具体而言，使用Reset-18的骨干，SVREP实现了$ \ textbf {65.9 \％} $的$ \ textbf {65.9 \％} $，以$ \ textbf {16.7} $ fps，在ICDAR2015上运行（视频）与$ \ textbf {8.6 \％} $提高的数据集比以前的最先进的方法。

大多数现有的视频文本发现基准测试专注于评估单一语言和具有有限数据的场景。在这项工作中，我们引入了大规模的双语，开放世界视频文本基准数据集（BovText）。 BovText有四个功能。首先，我们提供2,000多个具有超过1,75万多帧的视频，比现有最大数据集大25倍，其中包含录像中的附带文本。其次，我们的数据集涵盖了具有多种各种场景的30多个开放类别，例如Life VLog，驾驶，电影等。第三，为不同的代表提供了丰富的文本类型注释（即标题，标题或场景文本）视频中的意义。第四，BOVTEXT提供双语文本注释，以促进多种文化的生活和沟通。此外，我们提出了一个与变压器的端到端视频文本发现框架，被称为TransVtspotter，它通过简单但高效的关注的查询密钥机制解决了视频中的多东方文本。它将来自前一个帧的对象特征应用于当前帧的跟踪查询，并引入旋转角度预测以适合多大学实例。在ICDAR2015（视频）上，Transvtspotter以44.1％的Mota，9 FPS实现最先进的性能。 DataSet和TransVtspotter的代码可以在GitHub中找到：COM = Weijiawu = BovText和GitHub：Com = Weijiawu = Transvtspotter。

An oft-cited open problem of federated learning is the existence of data heterogeneity at the clients. One pathway to understanding the drastic accuracy drop in federated learning is by scrutinizing the behavior of the clients' deep models on data with different levels of "difficulty", which has been left unaddressed. In this paper, we investigate a different and rarely studied dimension of FL: ordered learning. Specifically, we aim to investigate how ordered learning principles can contribute to alleviating the heterogeneity effects in FL. We present theoretical analysis and conduct extensive empirical studies on the efficacy of orderings spanning three kinds of learning: curriculum, anti-curriculum, and random curriculum. We find that curriculum learning largely alleviates non-IIDness. Interestingly, the more disparate the data distributions across clients the more they benefit from ordered learning. We provide analysis explaining this phenomenon, specifically indicating how curriculum training appears to make the objective landscape progressively less convex, suggesting fast converging iterations at the beginning of the training procedure. We derive quantitative results of convergence for both convex and nonconvex objectives by modeling the curriculum training on federated devices as local SGD with locally biased stochastic gradients. Also, inspired by ordered learning, we propose a novel client selection technique that benefits from the real-world disparity in the clients. Our proposed approach to client selection has a synergic effect when applied together with ordered learning in FL.

Large language models can perform new tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior. Such prompts are typically hand engineered, but can also be learned with gradient-based methods from labeled data. However, it is underexplored what factors make the prompts effective, especially when the prompts are natural language. In this paper, we investigate common attributes shared by effective prompts. We first propose a human readable prompt tuning method (F LUENT P ROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a diverse distribution of effective and fluent prompts. Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of label words. Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks.

We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets.

Weakly supervised machine learning algorithms are able to learn from ambiguous samples or labels, e.g., multi-instance learning or partial-label learning. However, in some real-world tasks, each training sample is associated with not only multiple instances but also a candidate label set that contains one ground-truth label and some false positive labels. Specifically, at least one instance pertains to the ground-truth label while no instance belongs to the false positive labels. In this paper, we formalize such problems as multi-instance partial-label learning (MIPL). Existing multi-instance learning algorithms and partial-label learning algorithms are suboptimal for solving MIPL problems since the former fail to disambiguate a candidate label set, and the latter cannot handle a multi-instance bag. To address these issues, a tailored algorithm named MIPLGP, i.e., Multi-Instance Partial-Label learning with Gaussian Processes, is proposed. MIPLGP first assigns each instance with a candidate label set in an augmented label space, then transforms the candidate label set into a logarithmic space to yield the disambiguated and continuous labels via an exclusive disambiguation strategy, and last induces a model based on the Gaussian processes. Experimental results on various datasets validate that MIPLGP is superior to well-established multi-instance learning and partial-label learning algorithms for solving MIPL problems. Our code and datasets will be made publicly available.

Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 9 closed-set tasks and 7 open-set tasks demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better on dealing with rare patterns (word senses or facts), and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.

聚集的联合学习（FL）已显示通过将客户分组为群集，从而产生有希望的结果。这在单独的客户群在其本地数据的分布方面有显着差异的情况下特别有效。现有的集群FL算法实质上是在试图将客户群体组合在一起，以便同一集群中的客户可以利用彼此的数据来更好地执行联合学习。但是，先前的群集FL算法试图在培训期间间接学习这些分布相似性，这可能会很耗时，因为可能需要许多回合的联合学习，直到群集的形成稳定为止。在本文中，我们提出了一种新的联合学习方法，该方法直接旨在通过分析客户数据子空间之间的主要角度来有效地识别客户之间的分布相似性。每个客户端都以单一的方式在其本地数据上应用截断的奇异值分解（SVD）步骤，以得出一小部分主向量，该量提供了一个签名，可简洁地捕获基础分布的主要特征。提供了一组主要的主向量，以便服务器可以直接识别客户端之间的分布相似性以形成簇。这是通过比较这些主要向量跨越的客户数据子空间之间主要角度的相似性来实现的。该方法提供了一个简单而有效的集群FL框架，该框架解决了广泛的数据异质性问题，而不是标签偏斜的更简单的非iids形式。我们的聚类FL方法还可以为非凸目标目标提供融合保证。我们的代码可在https://github.com/mmorafah/pacfl上找到。